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In this experimental study, use of Computer Assisted Instruction ( CAI) followed 
by use of an Intelligent Tutoring System (CAI+ITS) was compared to the use of 
CAI (CAI only) in tutoring students on the topic of Algebraic Expression. Two 
groups of students participated in the study. One group of 32 students studied 
algebraic expression in a CAI learning environment, while the other group of 30 
students was in a CAI and ITS (CAI+ITS) environment. Before the experimental 
treatment began, subjects were given a pre-test on algebraic expression. A post- 
test was also given at the end of the study. The experimental treatment was 
administered in eight sessions with one hour per session. For the first stage of the 
study, both groups of subjects studied algebraic expression in a CAI environment. 
In the second stage, subjects from the CAI group continued with a tutoring session 
using the drill and practice section of the CAI package, whereas subjects from the 
CAI+ITS environment continued their learning using the ITS tutorial. The results 
of the study showed that there was a significant difference in the students’ 
achievement in algebraic expression between students who learned with CAI+ITS 
and who learned with CAI only as the delivery system. The findings of the study 
indicated that CAI+ITS was more effective in helping students learn algebraic 
expression as compared to using CAI alone. This study suggests that educators 
and software developers should focus on the development of ITS based learning 
tools or integrate ITS elements in courseware development rather than developing 
a mere CAI tool. 
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learning 
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INTRODUCTION 

While CAI may be somewhat effective in helping learners, it does not provide 
the same kind of personalised attention that a student would receive from a 
human tutor (Ester, 1994). In order to achieve a more efficient learning 
environment and to deliver the best learning process, research on ICT in 
mathematics education is focusing on the area of Artificial Intelligence (AI). 
Ester (1994) asserted that AI is able to prepare a more human-based interactive 
learning environment for students. A human based interactive learning 
environment is important because it involves students in active learning. 

Heffernan (2001) stated that as the techniques of AI become widely known and 
appreciated in the field of educational computing, AI with interests in education 
has also undergone changes in direction. He also stressed that the overall aim of 
developing AI is to enable the computer to be effective and act as a 
knowledgeable agent in the teaching and learning process. A major strand of 
research has been the design of the so-called Intelligent Tutoring Systems (ITS) 
which require knowledge representations to provide models of the subject 
domain, the learner capabilities, and the tutorial pedagogy (Heffernan, 2001). 
Canfield (2001) defined ITS as a system that is able to diagnose and adapt to 
students’ knowledge and skills. According to Canfield (2001), an ITS is able to 
provide precise feedback when mistakes are made and able to present new 
topics when the student is ready to learn. Intelligent tutoring systems are part of 
a new breed of instructional computer programs. 

The objective of an ITS is to provide a teaching process that adapts to the 
students’ needs by exploring and understanding the students’ special needs and 
interests (Kaplan & Rock, 1995). Research in the field of ITS has always had a 
strong focus on the development of comprehensive student models, based on 
the assumption that within a problem-solving context, learners’ thinking 
processes can be modelled, traced, and corrected using computers (Siemer- 
Matravers, 1999). 

According to McArthur, Eewis, and Bishay (1994), ITS attempt to capture a 
method of teaching and learning exemplified by a one-to-one human tutoring 
interaction. They further assert that one-to-one tutoring allows learning to be 
highly individualized and consistently yields better outcomes than other 
methods of teaching. Unlike previous CAI systems, ITS represent some of the 
knowledge and reasoning of good one-to-one human tutors, and consequently 
can coach in a much more detailed way than CAI systems. 



International Journal of Instruction, July 2008 • Vol.l, No. 2 




Chien, Md. Yunus, Ali, & Bakar 



27 



In relation to this, previous studies have indicated that students always face 
difficulty in learning algebraic expression in mathematics (Booth, 1984, 1988; 
Greeno, 1982; Kieran, 1988; Lins, 1990). According to Chick et al. (2001), 
students who have difficulties in solving mathematics problem such as 
algebraic-related mathematics problems usually have problems keeping up with 
classroom instruction. In an effort to help students overcome these learning 
problems, some studies have discovered the potential of self-paced CAI as a 
delivery system. 

Further exploration need to be conducted whether CAI, ITS or both CAI and 
ITS would benefit students in the learning. Studies have shown that use of ITS 
had shown improvement in student’s performance in algebra (Abidin and 
Hartley!, 1998; Moundridou and Virvou, 2002). Likewise, in Mohammad Naim 
and Finch’s (2001) study on the learning of Technical Education, they had also 
found that ITS was significantly more effective than conventional CAL 
Consistent result was also found in studies conducted by Koedinger et al. 
(1997). 

Thus, in order to find out which of the learning environments, the ITS or the 
CAI environment, is more suitable to help students in their learning of algebraic 
expression, this study attempts to investigate and compare the effectiveness of a 
CAI plus ITS (CAI-i-ITS) based approach and a CAI only approach in the 
learning of mathematics among secondary school students. 

The ITS systems in this study refers to the expert system, Ms Lindquist, that 
was presented based on dialog-based interaction with student. It was developed 
by Heffernan (2001). It is able to diagnose the learner’s action in order to infer 
the learner’s cognitive state, such as his or her level of knowledge or 
proficiency. Hence, the CAI-i-ITS intervention represents a combination of a 
CAI approach followed by an ITS tutorial in the learning of algebraic 
expression among the subjects in this study. 

METHOD 

This study adapted a pre-test post-test control group experimental design. The 
design of this study is represented in Figure 1 . In determining the effectiveness 
of ITS in helping students learn, this study involves independent variables 
which include two different combinations of delivery systems, CAI-i-ITS, and 
CAI only. The dependent variable is students’ achievement in algebraic 
expression. As illustrated in Figure 1, students were randomly assigned to the 
experimental group and control group (R). Both groups took the pre-test (Oi) 
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before the start of the experiment. The two groups later followed a three-hour 
learning session using the CAI courseware. The students were instructed to go 
through the tutorial section of the courseware. The researcher monitored the 
whole process to ensure that students were on the instructed tasks. Once both 
groups have learnt the content using the courseware, the experimental group 
were given the treatment, which is in the form of sessions using the ITS tutorial. 
On the other hand, the control group underwent further learning using the 
exercise and e-test sections of the CAI courseware. Upon completion of the 
treatment, both groups of students took the post-test (O 2 ). For the purpose of 
referring to the groups, the experimental group is named CAI-i-ITS group and 
the control as CAI only group. Pre-test on Algebraic Expression, administered 
before the experiment, was used as a covariate to control individual differences. 



Experimental 


R Oi X O2 


Control 


R Oi O2 



Figure 1: Experimental Design in the Study 



This study was conducted over a period of eight school days. Manson and 
Bramble (1997) pointed out that the longer the time spent, the greater the 
probability that something could influence the subjects’ environment that in 
turn would affect the results. Duration of eight school days was deemed 
appropriate to see effects of the experimental treatment. All the subjects in this 
study sat for the same pre-test and post-test. Thus, threats to internal validity 
such as maturation, instrumentation bias, history and regression were not 
serious threats to the internal validity of the study. 

The sample consisted of 62 respondents, approximately half of them were males 
(n=34), while the rest were females (n=28). The respondents were selected 
among Form One students (equivalent to Grade 7 in the United States) in a 
secondary school and their ages range between 12 to 13 year olds. Form one 
students were selected in this study because they had not learnt the topic on 
algebraic expression, which will only be taught in Form Two (Grade 8) 
Mathematics in Malaysia. Hence, the subjects in this study can be considered as 
not having had any knowledge on algebraic expression when this experimental 
study was administered. This is to ensure that the students did not have prior 
knowledge in the mathematics content used in this study. 

After getting the list of students for the two classes that participated in this 
study, the researcher randomly assigned the subjects into the two different 
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groups, namely, the CAI+ITS experimental group and the CAI-only control 
group. The school selected for the study was equipped with more than 100 
computers in its computer lahs and English is widely used in instruction. The 
school required all students to attend computer classes. This reduces any 
inconvenience in carrying out the study that might he due to students’ 
unfamiliarity with the use of computers in learning. During the time that the 
experiment was conducted, which was in 2003, Malaysia just shifted to the use 
of English in the teaching of mathematics, which was previously conducted in 
the Malaysian national language. However, during the transition period, most 
schools were not teaching fully in English as was determined hy the Ministry of 
Education. Therefore, the school selected also reduces the problem of students 
not understanding the ITS and CAI contents which were in English. 

INSTRUMENTATION 

Two equivalent sets of mathematics tests were prepared; a pre-test and post-test. 
A panel of mathematics teachers who had at least three years of teaching 
experience was consulted to validate the mathematics tests and to ensure that 
the items were equivalent. Reliahility of the mathematics tests were 0.87. 

Each of the tests comprised of 15 items on algebraic expression. The test items 
were based on the learning outcomes as stated in the syllabus of Eorm Two 
Mathematics. The test covered the concept of algebraic terms in two or more 
unknowns, computations involving multiplication and division of two of more 
terms, writing algebraic expressions for given situations using letters to 
represent unknowns and computations involving algebraic expressions. The 
tests were scored from 0 to 100. 

Subjects of the study underwent learning using the CAI courseware and/or the 
ITS system. The following section elaborates on the teaching materials used. 

CAI Courseware 

In Malaysia, each of the text books used for a particular school year must be 
supplemented with a courseware. The CAI courseware comes with the text 
book and is not for sale separately. Eor each school year, eight text books are 
approved and endorsed to be used in schools by the Textbook Division of the 
Ministry of Education. The CAI courseware selected for use in this study was 
the one developed by Pelangi Mindedge. Thus, a written permission was 
obtained from Pelangi Mindedge. Eor this study, the students were asked to go 
through the chapter on Algebraic Expressions. The chapter comprises of four 
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contents; concept of algebraic terms in two or more unknowns, computations 
involving multiplication and division of two or more terms, concept of algebraic 
expressions, and computations involving algebraic expressions. 

Tbe courseware contains six main menus, namely user guide, navigator, 
contents, e-test, e-journal, and multimedia gallery. To familiarize students with 
the courseware operations, they were encouraged to explore the user guide 
menu. The student selected the specified content they want to learn from the 
navigator or content menu. The chapter is also provided with the exercise 
corner, website links, simulator, glossary and media gallery. The exercise 
corner is meant to determine the student’s understanding of the concepts taught 
in the chapter, whilst the e-test section provides students with ten objectives 
questions that can be randomly selected from the database. Students can try the 
tutorial from e-test section as many times as they want. 

The website link menu provided extra information regarding the mathematics 
concept. Besides helping students to master the mathematics concept, the 
courseware provides a simulation game that allows users to learn mathematical 
concepts by experimenting with numbers. Users can type in numbers and select 
calculation mode. The answers will then be processed and displayed 
immediately. The glossary and media gallery sections help students to 
understand the meaning of mathematics terms used in the courseware. 

ITS Software 

The ITS-based algebra learning software used in this study was ‘Ms Lindquist’ 
created by Heffernan (2001). Permission to use this instrument was obtained 
from the author. This software is developed to guide students in learning 
algebra. To start using the program, Ms Lindquist requires students to key in 
their current level of algebraic knowledge. The selection of levels of algebraic 
knowledge, as given by the program are (i) one-operator problems; (ii) one- 
operator involving distance, rate, and time; (iii) two-operator linear forms; (iv) 
two-operator problems with some involving division and parentheses; and (v) 
three and four operator problems. Students need to choose their current level of 
algebraic knowledge before they can start the tutorial session with Ms 
Lindquist. 

After logging into the Ms Lindquist program and making the selection of their 
current level of algebraic knowledge, students were given an algebra problem 
and were asked to solve the expression to symbolize the problem. Once the 
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student obtained the correct answer, Ms Lindquist gave the second question 
based on its analysis of the way the student answered the first question. 

Throughout the learning process, Ms Lindquist provided a step-by-step guide to 
help the student to solve the entire algebraic problem. This is carried out by the 
dialog-based technique that is built in Ms Lindquist. By the end of the tutoring 
session, the students were given a feedback of their current level of algebraic 
knowledge. The system also provided suggestions to the student on ways to 
perform better in future study of algebra. 

In addition, students can leave the system anytime. The system will save all the 
interactions between the system and the student in the database. Hence, the 
teacher can check back the students’ performance throughout the entire tutoring 
session. The content of this ITS software focused on algebraic expression, 
guiding students to write algebraic expressions for given situations using letters 
to represent unknowns. The researcher had showed the content of the ITS 
software to panel teachers who developed the test items to ensure that the test 
items are equivalent to the content in the ITS used in the study. This is to ensure 
that the coverage of contents in Ms Lindquist is similar to the coverage of 
contents for the topic ‘Algebraic Expression’ in the Malaysian Form Two 
(equivalent to United States Grade 8) Mathematics syllabus. 

PROCEDURES OE STUDY 

The students’ process of learning algebraic expression using the CAI and 
CAI-i-ITS approaches is explained in Figure 2. A pre-test was administered 
before the experimental session began. Both groups of students, the 
experimental and control group, were provided a session to learn using the CAI 
courseware selected. There were asked to go through the content on 
introduction of algebraic terms and followed by algebraic terms in two or more 
unknowns. After going through the introduction on the concept of algebraic 
expression, students were asked to go through the section on computations 
involving multiplication and division of two or more terms. 

Once both groups have gone through the learning session using the CAI 
courseware, the control group was assigned to a CAI-based tutorial whereas the 
other group sat for the ITS-based tutorial. Students who proceeded to the e-test 
section of the CAI courseware were required to answer a series of algebraic 
expression exercises. Based on the students’ performance, the courseware 
provided feedback such as ‘poor’, ‘average’, or ‘good’. 
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Those who proceeded with the ITS were engaged in a dialogue-hased tutorial 
environment with Ms Lindquist. Based on the students’ responses, the ITS 
provided feedhack from time to time to guide them in understanding the 
concept of algebraic expression. After five hours of tutorial lessons, both groups 
of students were then given the post-test on the topic 'algebraic expression’. 




Figure 2: Procedures of Study 
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FINDINGS 

Table 1 shows data for the two groups of respondents. The mean score for pre- 
test of the experimental group (CAI-i-ITS group) was 27.60, while the post-test 
for this group was 51.36. Paired-sample t-test among respondents in CAI -i-ITS 
group for pre-test and post-test (t (29) = -9.47, p < .05) indicated that there was 
significant differences between pre-test and post-test achievements. The mean 
gain score for this group of respondents was 23.76. 

Whereas for the control group (CAI only group), mean score of the pre-test for 
the respondents was 22.11 and the post-test was as 34.33. A paired sample t-test 
(t (31) = -13.95, p < .05) indicated that there was a significant difference 
between pre-test and post-test achievements. The mean gain score for CAI only 
group was 12.22. Thus, both groups of respondents achieved better results in 
the post-test as compared to the pre-test. However the CAI-i-ITS experimental 
group had shown a greater increase in scores between the post-test and the pre- 
test. 



Table 1: Pre-test and Post-test for CAI-tITS and CAI only approach 



Group 


n 


Pre- 

test 


Post- 

test 


Mean 

Gain 

Score 


t 


df 


Sig 


Experimental Group 

(CAMTS) 


30 


27.60 


51.36 


23.76 


-9.47 


29 


.00 


Control Group (CAI only) 


32 


22.11 


34.33 


12.22 


-13.95 


31 


.00 



The assumption for homogeneity of regression slopes was tested to check the 
interaction between the independent variable (delivery system) and the 
covariate (pre-test). Table 2 indicated that there is no significant interaction 
among the two variables (F (1,58) = 154.9, p < 0.05). Thus, assumption of 
homogeneity of regression slopes is not violated. This indicated that there is no 
interaction between the pre-test and delivery system on students’ achievement. 



Table 2: Homogeneity of regression test for gain scores for different delivery 
systems 



Source 


df 


Mean Square 


F Sig. 


Between group 

(Delivery System and Pre-test) 


1 


154.9 


2.170 .15 


Within group 


58 


71.40 
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A one-way between-groups analysis of covariance (ANCOVA) was conducted 
to test the effectiveness of the delivery systems in the learning of algebraic 
expression between the experimental and the control groups. The independent 
variable was the type of delivery system; CAI-i-ITS and CAI only. The 
dependent variable was the gain score on the Algebraic Expression tests. 

The preliminary checks were conducted to ensure that there was no violation of 
assumptions of normality, linearity, homogeneity of variances, homogeneity of 
regression slopes, and reliable measurement of the covariate. Levene’s test 
showed that result was not significant (p < 0.05), thus indicated that variance of 
the dependent variable is equal between the groups. 

The result of the ANCOVA is shown in Table 3. As indicated in the table, there 
was a statistically significant difference between the two delivery systems on 
gain scores in the Algebraic Expression tests [E (1,59) = 26.03, p < .05]. Thus, 
when pre-test scores were set as a covariate, it is shown that gain score is 
influenced by the delivery systems. Significant difference was established for 
students’ achievement in the learning of algebraic expression between students 
who use CAI-tlTS and students who use CAI only as their delivery system. 

Table 3: ANCOVA of Respondent’s Gain Scores for Different Delivery 
Systems 



Source 


df 


Mean Square F 


Sig. 


Between Group 
(Delivery System) 


1 


1895 26.03 


.00 


Within group 


59 


72.81 





DISCUSSION 



Einding of this study showed that the CAI-i-ITS approach was significantly more 
effective than the CAI only approach on students’ achievement in the learning 
of algebraic expression. This finding provides supporting evidence on the 
benefits of using CAI-i-ITS in the learning of mathematics. The findings are 
consistent to those of Moundridou and Virvou (2002), Abidin and Hartleyt 
(1998), Mohammad Naim and Einch (2001) and Koedinger et al. (1997). 

Hence, CAI-i-ITS is a better aid in guiding learning as compared to CAI-only 
approach. This is due to the powerful tools in ITS which provide useful 
feedbacks to learners. The system evaluates student’s answer by comparing the 
given answer to the answer provided by the students into the system. ITS will 



International Journal of Instruction, July 2008 • Vol.l, No. 2 




Chien, Md. Yunus, Ali, & Bakar 



35 



then generate explanations based on its evaluation of the given answer. As a 
result, students can construct new knowledge from the feedbacks provided by 
the system (Siemer-Matravers, 1 999). 

Earlier works had also highlighted on the inefficiency of CAI courseware in 
helping students to learn. Piccolo et al. (2001) and Jain and Getis’s studies 
(2003) reported that respondents in CAI group did not show any significant 
differences in their achievement as compared to the traditional conventional 
classroom group. Respondents in the CAI group did not perform as well as 
respondents from the other group in their post-test. Study by Fuchs et al. (2002) 
showed that computer-assisted practice added little value beyond tutoring and 
yielded only moderate effects on story problems and real-world problem 
solving. Similarly, Wang and Sleeman (1993) reported there were no 
differences in performance and retention level scores between CAI and 
conventional instruction. Thus, to add to the cost-effectiveness of CAI, other 
elements need to be incorporated to ensure that it will indeed be more effective 
than traditional face-to-face instruction. 

CONCLUSION 

Results of this study revealed that CAI-i-ITS has potential in mathematics 
learning via computer applications. The ITS approach appears to have value as 
an instructional tool for mathematics learning. The fact that these students 
achieved better results in the CAI-i-ITS approach indicates that ITS is viable 
instructional option. The one-to-one tutoring function in ITS enables it to adapt 
tutoring strategies according to the needs of the individual student. Thus, 
educators can spend more time to guide weaker students while others learn via 
ITS tutorials. ITS does not attempt to change the process of education in any 
radical way but it recognises the strengths of a human teacher and remove 
teacher’s burden in teaching (Wilkinson-Riddle and Patel, 1998). 

CAI systems have been developed for a long time and there are various types of 
CAI systems being used for guiding learning. In Malaysia, most of the learning 
tools provided are still in the form of a CAI-based tutorial. However, tools such 
as FunctionLab (Abidin and Hartley, 1998), WEAR (Moundridou and Virvou, 
2002) and Animate (Nathan et al., 1992) have been widely used in other 
countries. Based on the findings of this study and other studies that focused on 
the effectiveness of ITS, it is timely that elements of ITS are incorporated in the 
softwares developed for Malaysian students. This would certainly provide 
opportunities for students to get personalized tutoring in learning mathematics. 
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